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DETAILED ACTION 

I. Response to Arguments 

Applicant's response to the last Office Action, mailed 30 July 2008 has 
been entered and made of record. 

Claims 1-3, 5-11, 13-28, and 32-41 are pending in the application. 

Applicant's amendments to independent claims 1, 11, 22 and 38 are 
sufficient to overcome the rejection under 35 U.S.C. §101. Thus, this 
rejection is withdrawn. 

Applicant's arguments filed 23 July 2009 have been fully considered but 
they are not persuasive. 

Applicant's argument is that "Seago does not teach modeling an object at 
all where the object occupies a field of view greater than 180 degrees," 
and that "Seago's method [of modeling an object] is inapplicable to the 
objects that occupy a field of view of more than 180 degrees (Remarks: 
14)," for the following reasons: 

• "The input to Seago's method is a 2-D perspective image or a group of 
2-D images displaying the object to be modeled (See, e.g., Seago, col. 
11, lines 45-47) (Remarks: 13)." 

• "Seago determines vanishing points for the planar faces of the object 
using parallel lines on the object and then derives a 3D coordinate 
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system for the object using the vanishing points (See, Seago, steps 40 

to 48, fig. 2.) (Remarks: 13)." 
• "The features of the object are then modeled (See, Seago, step 50, fig. 

2 and figs. 9-10.) (Remarks: 13)." 
So whether or not the method of Seago can be applied to input images 
depends on whether or not "a 2-D perspective image or a group of 2-D 
images displaying the object to be modeled" are used as input. Szeliski 
teaches, "the user creates a small mosaic using a planar perspective 
motion model... (Szeliski: Col. 7, Lns. 10-13)." In other works, Szeliski is 
also using 2-D (planar) images that include perspective. As such, one of 
ordinary skill would be able to apply the method of Seago, which takes 2- 
D planar images, determines vanishing points to derive a 3D coordinate 
system, and models the objects within the images, to the teachings of 
Szeliski which also include use of 2D perspective images. 

According to KSR Int'l Co. v. Teleflex, Inc., 127 S. Ct. 1727, 1739 (2007) 
at 1396, "if a technique has been used to improve one device, and a 
person of ordinary skill in the art would recognize that it would improve 
similar devices in the same way, using the technique is obvious unless 
its actual application is beyond his or her skill." As established above, 
both Seago and Szeliski use 2D perspective images and one of ordinary 
skill would recognize how to create object models from the 2D 
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perspective images of Szeliski using the method of Seago. Examiner 
notes that the perspective information required to perform the method of 
Seago is not eliminated in Szeliski, regardless of whether the objects in 
the images occupy a field of view greater than 180 degrees. Therefore, 
Examiner maintains her position that the combination of Szeliski and 
Seago would have been obvious and teaches limitations for which they 
are cited. 

II. Claim Rejections - 35 USC § 1 03 

The following is a quotation of 35 U.S. C. 103(a) which forms the basis for 

all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or 
described as set forth in section 102 of this title, if the differences between the subject matter 
sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the 
art to which said subject matter pertains. Patentability shall not be negatived by the manner 
in which the invention was made. 

A. Claims 22-28, 32-35, 38, and 40-41 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Szeliski et al. (US Patent Number 6,157,747, herein 
referred to as Szeliski) in view of Seago (US Patent Number 5,990,900). 

Regarding independent claim 22, Szeliski teaches a computerized method 
for projecting texture information onto an object within an image 
panorama (See Szeliski; Fig. 2B, Item 270; Col. 27, Lns. 62-66), the 
method comprising: 
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• Using the computer receiving instructions from a user identifying a 
three-dimensional geometric surface within an image panorama (See 
Szeliski: Col. 27-28, Lns. 62-2, respectively, wherein, "The shape of 
the model and the embedding of each face into texture space are left 
up to the user. This choice can range from something as simple as a 
cube with six separate texture maps, to something as complicated as 
a subdivided dodecahedron, or even a latitude-longitude tessellated 
globe."), 

o The image panorama containing an object having one or more 
textures (See Szeliski: Col. 5, Lns. 27-35), 

o The object occupying a field of view of more than 180 degrees in 
the panorama (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
as large a field of view as desired, e.g., a complete environment 
map."); 

• Determining a directional vector from the three-dimensional geometric 
surface (See Szeliski: Fig. 15; Col. 22, Lns. 4-31); 

• Using the computer creating a geometric model of the image 

panorama based at least in part on the three-dimensional geometric 
surface and the directional vector (See Szeliski: Col. 7, Lns. 29-33, "By 
mapping the mosaic onto an arbitrary texture-mapped polyhedron 
surrounding the origin, the virtual environment is viewed using 
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standard 3D graphics viewers and hardware (block 160) without 
requiring special purpose players." In Col. 22, Lns. 4-31, Szeliski 
teaches the direction vectors, which are used to align the images of 
the panoramic image. Thus, the geometric model of the visual scene is 
dependent on the geometric shapes of the surfaces of the polyhedron 
to be texture mapped, and the texture map applied to the surfaces of 
the polyhedron, which is the panoramic mosaic determined using the 
direction vectors.); and 

• Applying the one or more textures to the object in the image 
panorama based on the geometric model (See Szeliski: Fig. 2B, Item 
270; Col. 28, Lns. 13-15, wherein Szeliski teaches, "...efficiently 
computing texture map color values for any geometry and choice of 
texture map coordinates."). 

Szeliski does not expressly suggest: 

• Creating a geometric model includes identifying at least one boundary 

of the object and 

• Using the identified boundary to associate geometry information with 
the object, 

o The geometry information comprising 3-D coordinates describing 
the position and orientation of the object boundary in a reference 
coordinate system. 
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Seago is cited for teaching, "Line creation can be performed automatically 
by image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges (See Seago: Col. 4, Line 
65 - Col. 5, Line 1, respectively)," which correspond to identifying at least 
one boundary of the object. Seago also teaches, "Next, at blocks 50, 
shapes or polygons that define the selected object's sides, and plane 
indexes are determined based on user designated vertices or vanishing 
lines at significant features of the selected object (See Seago: Fig. 2, Item 
50; Col. 5, Lns. 35-38)," which corresponds to using the boundary to 
associate geometry information with the object. Finally, Seago is cited for 
teaching, "Once all the polygons and plane indexes of a selected object 
have been determined, a three-dimensional object oriented within the 
selected object's three-dimensional coordinate space is determined (See 
Seago: Fig. 2, Item 54; Col. 5, Lns. 50-55)," which corresponds to the 3D 
coordinates describing the position and orientation of the object 
boundary in the reference coordinate system. 

It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Szeliski, because Seago is in the same field of 
endeavor of generating three-dimensional computer graphics using image 
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capture devices; and Seago expressly suggests that this method of object 
extraction produces accurate three-dimensional objects more efficiently 
than conventional systems that try to extract the three-dimensional 
objects using analytical mathematical interpretations and orthogonal 
image analysis (See Seago: Col. 11, Lns. 49-55). Moreover, Applicant has 
not alleged any reason in support of why extracting a 3D object from a 
2D image panorama would be surprising or unexpected to one skilled in 
the pertinent art, or why there might have been some unexpected 
advantage in doing so. Applicant has provided no evidence to show that 
extracting 3D objects from 2D images with a view exceeding 180° was 
"uniquely challenging or difficult for one of ordinary skill in the art," (see 
Leapfrog Enters., Inc. v. Fisher-Price, Inc., 485 F.3d 1157, 1162 (Fed. Cir. 
2007) (citing KSR, 127 S. Ct. at 419). 

Regarding independent claim 32, the Szeliski teaches a system for 
creating a three dimensional model from a plurality of image panoramas, 

the system comprising: 

• Means for receiving the image panoramas representing a visual scene 
(See Szeliski: Col. 27, Lns. 34-37, "Once a complete panoramic mosaic 
has been constructed, it is necessary to convert the set of input 
images and associated transforms into one or more images which can 
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be quickly rendered or viewed.") having an object (See Szeliski: Fig. 6, 
Item 610; Col. 11, Lns. 16-21); 

o The object occupying a field of view of more than 180 degrees in 
the panoramas (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 

images as needed, image mosaics can be constructed which cover 
as large a field of view as desired, e.g., a complete environment 
map."), 

• Means for allowing a user to interact with the system to determine a 
directional vector for each image panorama (See Szeliski: Col. 25, Lns. 

64-66); 

• Means for aligning the image panoramas relative to each other (See 
Szeliski: Fig. 17; Col. 23, Lns. 10-12, wherein minimizing the 
differences between ray directions aligns the images.). 

Szeliski does not expressly suggest: 

o Means for creating a three dimensional model from the aligned 

panoramas, wherein: 

o Creating a three dimensional model includes identifying at least 

one boundary of the object and 
o Using the identified boundary to associate geometry information 

with the object includes associating geometry information with the 

object. 
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■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system. 

Seago is cited for teaching: 

o Means for creating a three dimensional model from the aligned 
panoramas (See Seago: Fig. 2; Col. 4, Lns. 23-25, "FIG. 2 illustrates 

the process by which the image converting system 20 generates a 
three-dimensional model or object from a single two-dimensional 
image."), wherein: 

o Creating a three dimensional model includes identifying at least 
one boundary of the object (See Seago: Col. 4, Line 65 - Col. 5, Line 
1, respectively, "Line creation can be performed automatically by 
image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges.") and 

o Using the identified boundary to associate geometry information 
with the object includes associating geometry information with the 
object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 35-38, "Next, at 
blocks 50, shapes or polygons that define the selected object's 
sides, and plane indexes are determined based on user designated 
vertices or vanishing lines at significant features of the selected 
object."). 
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■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system (See Seago: Fig. 2, Item 54; 
Col. 5, Lns. 50-55, "Once all the polygons and plane indexes of 
a selected object have been determined, a three-dimensional 
object oriented within the selected object's three-dimensional 
coordinate space is determined."). 

It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Szeliski, because Seago is in the same field of 
endeavor of generating three-dimensional computer graphics using image 
capture devices; and Seago expressly suggests that this method of object 
extraction produces accurate three-dimensional objects more efficiently 
than conventional systems that try to extract the three-dimensional 
objects using analytical mathematical interpretations and orthogonal 
image analysis (See Seago: Col. 11, Lns. 49-55). Moreover, Applicant has 
not alleged any reason in support of why extracting a 3D object from a 
2D image panorama would be surprising or unexpected to one skilled in 
the pertinent art, or why there might have been some unexpected 
advantage in doing so. Applicant has provided no evidence to show that 
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extracting 3D objects from 2D images with a view exceeding 180° was 
"uniquely challenging or difficult for one of ordinary skill in the art," (see 
Leapfrog Enters., Inc. v. Fisher-Price, Inc., 485 F.3d 1157, 1162 (Fed. Cir. 
2007) (citing KSR, 127 S. Ct. at 419). 

Independent claim 38 differs from independent claim 32 primarily in that 
claim 38 is directed toward the computerized method implemented on 
the system of claim 32. Claim 38 also differs from claim 32 in that it 
requires creating a three dimensional model of the visual scene from the 
image panorama using the reference coordinate system. Seago is cited for 
teaching, "Next, at block 42, an object contained within the digital image 
is selected for conversion into a three- dimensional object, and the 
selected object's orientation, or natural coordinate system, is 
approximately determined (See Seago: Fig. 2, Item 42; Col. 4, Lns. 46-49, 
wherein the reference coordinate system is the natural coordinate 
system.)." Therefore, the rationale of claim 32 is applied to claim 38. 

Regarding claim 24, as it depends from claim 22, Szeliski teaches the 
three-dimensional geometric surface is one of a floor, a wall, or a ceiling 
(See Szeliski: Col. 27, Lns. 64-67, wherein the model can be a cube with 
six separate texture maps for each surface. Using an appropriate 
environment map would cause the top surface to be a ceiling, the bottom 
surface to be a floor, etc.). 
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Regarding claim 25, as it depends from claim 22, Szeliski teaches the 
directional vector is orthogonal to the planar surface (See Szeliski: Fig. 
15; Col. 22, Lns. 4-6). 

Regarding claim 26, as it depends from claim 22, Szeliski teaches the 
geometric model comprises depth information (See Szeliski: Figs. 27 and 
30; Col. 28, Lns. 29-33). 

Regarding claim 27, as it depends from claim 22, Szeliski teaches the 
texture information comprises color information (See Szeliski: Col. 28, 
Lns. 13-18). 

Regarding claim 28, as it depends from claim 22, Szeliski teaches the 
texture information comprises luminance information (See Szeliski: Fig. 
2B; Col. 5, Lns. 27-30, wherein luminance is inherent to environment 
maps and texture maps.). 

Regarding claim 33, as it depends from claim 32, Szeliski teaches a 
system wherein the input images comprise two-dimensional images (See 
Szeliski: Figs. 3-4; Col. 9, Lns. 56-58, wherein a camera 310 having its 
optical center fixed at point C (FIG. 3) captures a sequence of 2D still 
images lo, Ii, h, h...). 
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Regarding claim 34, as it depends from claim 32, Szeliski teaches the 
input images comprise three-dimensional images including geometry 
information (See Szeliski: Figs. 3-4 and 6; Col. 9, Lns. 56-62, wherein the 
camera captures a sequence of 2D still images (lo, Ii, h, h.) as it pans, 
the center rays of these images being focused on 3D points (Po, Pi, P2, 
P3...) at a focal length f from the optical center point C. The points Pi are 
defined relative to a fixed 3D world coordinate system (Px, Py, Pz). Since 
the three-dimensional images correspond to the two-dimensional images 
which include depth information in the form of focal length. The 
geometry information is the inverted V shape shown in both figures 4 
and 6.). 

Regarding claim 35, as it depends from claim 32, Szeliski teaches 
aligning the image panoramas according to instructions received from a 
user (See Szeliski: Col. 27, Lns. 64-66). 

Regarding claims 40 and 41, as they depend from claims 22 and 38, 
respectively, Szeliski teaches the object is a room and the at least one 
boundary of the object is the intersection of a wall of the room with the 
floor (See Szeliski: Col. 27, Lns. 64-67, wherein the model can be a cube 
with six separate texture maps for each surface. Using an appropriate 
environment map would cause the top surface to be a ceiling, the bottom 
surface to be a floor, etc.). 
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B. Claims 1 1 , 13-21 , 23, and 36-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of Szeliski and Seago, and in further view of 
Blank (US Patent Number 5,469,536). 

Regarding independent claim 1 1 , Szeliski teaches a computerized method 
of interactively editing objects in a panoramic image, the method 
comprising: 

• Receiving an image panorama representing a visual scene (See 
Szeliski: Col. 27, Lns. 34-37, "Once a complete panoramic mosaic has 
been constructed, it is necessary to convert the set of input images 
and associated transforms into one or more images which can be 
quickly rendered or viewed."), 

o The image panorama having an object (See Szeliski: Fig. 6, Item 
610; Col. 11, Lns. 16-21) and a point source (See Szeliski: Figs. 3-4 
and 6; Col. 9, Lns. 56-62, wherein the camera captures a sequence 
of 2D still images (lo, Ii, h, h.) as it pans, the center rays of these 
images being focused on 3D points (Po, Pi, Pa, Pa...) at a focal length 
/from the optical center point C"), 

o The object occupying a field of view of more than 180 degrees in 
the panorama (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
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as large a field of view as desired, e.g., a complete environment 
map."); 

Szeliski does not expressly suggest: 

• Using the computer creating a three dimensional model of the visual 
scene using features of the visual scene and the point source, 
wherein: 

o Creating a three dimensional model includes identifying at least 

one boundary of the object and 
o Using the identified boundary to associate geometry information 

with the object, 

■ The geometry information comprising 3-D coordinates 

describing the position and orientation of the object boundary 
in a reference coordinate system. 

Seago is cited for teaching: 

• Creating a three dimensional model of the visual scene (See Seago: 
Fig. 2; Col. 4, Lns. 23-25, "FIG. 2 illustrates the process by which the 
image converting system 20 generates a three-dimensional model or 
object from a single two-dimensional image.") using features of the 
visual scene and the point source (See Seago: Fig. 2, Item 40; Col. 4, 
Lns. 34-37, "First, at block 40, a digitized two-dimensional 
perspective image of one or more objects is retrieved from memory 32 
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or a digital image input device 28 and displayed on the display device 
24," wherein the features of the visual scene are the one or more 
objects and the point source is indicated by the perspective of the 
image.), wherein: 

o Creating a three dimensional model includes identifying at least 
one boundary of the object (See Seago: Col. 4, Line 65 - Col. 5, Line 
1, respectively, "Line creation can be performed automatically by 
image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges.") and 

o Using the identified boundary to associate geometry information 
with the object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 35-38, 
"Next, at blocks 50, shapes or polygons that define the selected 
object's sides, and plane indexes are determined based on user 
designated vertices or vanishing lines at significant features of the 
selected object."), 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in a reference coordinate system (See Seago: Fig. 2, Item 54; 
Col. 5, Lns. 50-55, "Once all the polygons and plane indexes of 
a selected object have been determined, a three-dimensional 
object oriented within the selected object's three-dimensional 
coordinate space is determined."). 
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It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Szeliski, because Seago is in the same field of 
endeavor of generating three-dimensional computer graphics using image 
capture devices; and Seago expressly suggests that this method of object 
extraction produces accurate three-dimensional objects more efficiently 
than conventional systems that try to extract the three-dimensional 
objects using analytical mathematical interpretations and orthogonal 
image analysis (See Seago: Col. 11, Lns. 49-55). Moreover, Applicant has 
not alleged any reason in support of why extracting a 3D object from a 
2D image panorama would be surprising or unexpected to one skilled in 
the pertinent art, or why there might have been some unexpected 
advantage in doing so. Applicant has provided no evidence to show that 
extracting 3D objects from 2D images with a view exceeding 180° was 
"uniquely challenging or difficult for one of ordinary skill in the art," (see 
Leapfrog Enters., Inc. v. Fisher-Price, Inc., 485 F.3d 1157, 1162 (Fed. Cir. 
2007) (citing KSR, 127 S. Ct. at 419). 

The combination of Szeliski and Seago does not expressly suggest: 

• Using the computer receiving an edit to the object in the panorama; 
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• Using the computer transforming the edit relative to a viewpoint 
defined by the point source; and 

• Projecting the transformed edit onto the object. 

Blank is cited for teaching trimming, which allows the user to trim off 
any undesired edges of an object to reveal the background below (See 
Blank: Col. 47, Lns. 11-20). It would have been obvious to one of 
ordinary skill in the art, at the time the invention was made, to have 
allowed modification of the colors within an image, as taught by Blank, 
to manipulate the colors within the images used to create the panoramic 
image mosaics, as taught by Szeliski, as modified by Seago, because it 
would enable the user to quickly and efficiently modify or enhance the 
appearance of an image to desired goal (See Blank: Col. 6, Lns. 23-28). 

Independent claim 36 differs from independent claim 1 1 primarily in that 
claim 36 is directed toward the system for implementing the 
computerized method of claim 1 1 . Claim 36 also differs in that it requires 
creating a three dimensional model includes identifying at least one 
boundary of the object and one or more interactive editing tools for 
providing an edit to the selected object. Seago teaches, "FIG. 2 illustrates 
the process by which the image converting system 20 generates a three- 
dimensional model or object from a single two-dimensional image (See 
Seago: Fig. 2; Col. 4, Lns. 23-25)," wherein "... at block 42, an object 
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contained within the digital image is selected for conversion into a three- 
dimensional object, and the selected object's orientation, or natural 
coordinate system, is approximately determined (See Seago: Fig. 2, Item 
42; Col. 4, Lns. 46-49)," which corresponds to creating a three 
dimensional model includes identifying at least one boundary of the 
object. Blank is cited for teaching function calls, which correspond to the 
interactive editing tools for editing the selected object (See Blank: Col. 2 1 , 
Lns. 18-23). Thus, the rationale of independent claim 11 is applied to 
independent claim 36. 

Regarding claim 13, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to color information associated with the object (See 
Blank: Col. 6, Lns. 29-47). 

Regarding claim 14, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to alpha information associated with the object (See 
Blank: Fig. 11; Col. 20, Lns. 61-66). 
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Regarding claim 15, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to depth information associated with the object (See 
Blank: Col. 13, Lns. 8-16). 

Regarding claim 16, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
receiving an edit to geometry information associated with the object (See 
Blank: Col. 47, Lns. 11-20). 

Regarding claim 17, as it depends from claim 11, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
providing a user with an interactive drawing tool that specifies edits for 
the object and receiving the edits made by the user using the interactive 
drawing tool (See Blank: Col. 21, Lns. 18-23). 

Regarding claims 18 and 37, as they depend from claims 17 and 36, 
respectively, the rationale of claim 11 is incorporated herein. The 
combination of Szeliski, Seago, and Blank substantially teach the 
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invention as claimed. Specifically, Blank teaches the interactive drawing 
tool is a depth chisel tool (See Blank: Col. 47, Lns. 1 1-20). 

Regarding claim 19, as it depends from claim 17, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
the interactive drawing tool specifies a selected value for depth for the 
object (See Blank: Col. 22, Lns. 35-62). 

Regarding claim 20, as it depends from claim 17, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
the interactive drawing tool incrementally adds to the depth for the object 
(See Blank: Col. 34, Lns. 8-11). 

Regarding claim 21, as it depends from claim 17, the rationale of claim 
1 1 is incorporated herein. The combination of Szeliski, Seago, and Blank 
substantially teach the invention as claimed. Specifically, Blank teaches 
the interactive drawing tool incrementally subtracts from the depth for 
the object (See Blank: Col. 34, Lns. 22-26). 

The rationale of claim 17 is applied to claim 23. 
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C. Claims 1-3, 5-6, 8-10 and 39 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Szeliski in view of Luken (US Patent Number 5,923,334), and 
in further view of Seago. 

Regarding independent claim 1, Szeliski teaches a computerized method 
for creating a three dimensional model from image panoramas, the 
method comprising: 

• Receiving at a computer a plurality of image panoramas representing 
a visual scene (See Szeliski: Col. 27, Lns. 34-37, "Once a complete 
panoramic mosaic has been constructed, it is necessary to convert the 
set of input images and associated transforms into one or more 
images which can be quickly rendered or viewed.") and having an 
object (See Szeliski: Fig. 6, Item 610; Col. 11, Lns. 16-21), 

o The object occupying a field of view of more than 180 degrees in 
the panoramas (See Szeliski: Col. 5, Lns. 7-9, "By taking as many 
images as needed, image mosaics can be constructed which cover 
as large a field of view as desired, e.g., a complete environment 
map."); 

• Using the computer, determining a directional vector for the image 
panorama, the directional vector indicating an orientation of the 
visual scene with respect to a reference coordinate system (See 
Szeliski: Fig. 15; Col. 22, Lns. 4-31); 
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• Transforming the image panoramas such that the directional vectors 
are substantially aligned relative to the reference coordinate system 
(See Szeliski: Fig. 15; Col. 22, Lns. 2-4, "FIG. 15 shows the 
adjustment of the bundle of rays xjk so that they converge to xj."); 

• Aligning the transformed image panoramas to each other (See 
Szeliski: Fig. 17; Col. 23, Lns. 10-12, wherein minimizing the 
differences between ray directions aligns the images.). 

Szeliski does not expressly suggest determining a directional vector for 
each image panorama. Luken is cited for teaching eight direction vectors 
D0-D7 associated with six rectangular images mapped to the inside of an 
octahedron (See Luken: Figs. 7-10, 14 and 17; Col. 7, Lns. 28-36, 
wherein it is determined which of the six rectangular images is 
intersected by one of the eight direction vectors.). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have used direction vectors for each image within an 
environment map that is mapped to the sides of an octahedron, as 
taught by Luken, with the three-dimensional model environment map of 
the visual scene, as taught by Szeliski, because Luken: (1) is directed to 
the same problem of using polyhedral environment maps to create and 
view three dimensional images from data representing multiple views of a 
scene; (2) is in the same field of endeavor of image processing systems; 
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and (3) Luken expressly suggests that the direction vectors provide an 
efficient system for generating and viewing three-dimensional panoramic 
images based environment maps, and offer an improved level of 
interactive graphical feedback (See Luken: Col. 3, Lns. 5-8). 

The combination of Szeliski and Luken does not expressly suggest: 

• Using the computer, creating a three dimensional model of the visual 
scene from the transformed and aligned image panoramas using the 
reference coordinate system, wherein 

o Creating a three dimensional model includes identifying at least 

one boundary of the object and 
o Using the identified boundary to associating associate geometry 

information with the object, 

■ The geometry information comprising 3-D coordinates 
describing the position and orientation of the object boundary 
in the reference coordinate system. 

Seago is cited for teaching: 

• Using a computer, creating a three dimensional model of the visual 
scene (See Seago: Fig. 2; Col. 4, Lns. 23-25, "FIG. 2 illustrates the 
process by which the image converting system 20 generates a three- 
dimensional model or object from a single two-dimensional image.") 
from the transformed and aligned image panoramas using the 
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reference coordinate system (See Seago: Fig. 2, Item 42; Col. 4, Lns. 
46-49, "Next, at block 42, an object contained within the digital image 
is selected for conversion into a three-dimensional object, and the 
selected object's orientation, or natural coordinate system, is 
approximately determined," wherein the reference coordinate system 
is the natural coordinate system.), wherein 

o Creating a three dimensional model includes identifying at least 
one boundary of the object (See Seago: Col. 4, Line 65 - Col. 5, Line 
1, respectively, "Line creation can be performed automatically by 
image analysis software, which determines edges of objects and 
creates lines overlapping the determined edges.") and 

o Using the identified boundary to associating associate geometry 
information with the object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 
35-38, "Next, at blocks 50, shapes or polygons that define the 
selected object's sides, and plane indexes are determined based on 
user designated vertices or vanishing lines at significant features of 
the selected object."), 

■ The geometry information comprising 3-D coordinates 

describing the position and orientation of the object boundary 
in the reference coordinate system (See Seago: Fig. 2, Item 54; 
Col. 5, Lns. 50-55, "Once all the polygons and plane indexes of 
a selected object have been determined, a three-dimensional 
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object oriented within the selected object's three-dimensional 
coordinate space is determined."). 

It would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used the three-dimensional object 
editing abilities of Seago, to modify the three-dimensional environment 
map, as taught by Szeliski, as modified by Luken, because Seago is in 
the same field of endeavor of generating three-dimensional computer 
graphics using image capture devices; and Seago expressly suggests that 
this method of object extraction produces accurate three-dimensional 
objects more efficiently than conventional systems that try to extract the 
three-dimensional objects using analytical mathematical interpretations 
and orthogonal image analysis (See Seago: Col. 11, Lns. 49-55). 
Moreover, Applicant has not alleged any reason in support of why 
extracting a 3D object from a 2D image panorama would be surprising or 
unexpected to one skilled in the pertinent art, or why there might have 
been some unexpected advantage in doing so. Applicant has provided no 
evidence to show that extracting 3D objects from 2D images with a view 
exceeding 180° was "uniquely challenging or difficult for one of ordinary 
skill in the art," (see Leapfrog Enters., Inc. v. Fisher-Price, Inc., 485 F.3d 
1157, 1162 (Fed. Cir. 2007) (citing KSR, 127 S. Ct. at 419). 
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Regarding claim 2, as it depends from claim 1, Szeliski teaches the 
directional vector is determined based, at least in part, on instructions 
identifying elements of the image panoramas received from a user (See 
Szeliski: Col. 8, Lns. 30-32 and Col. 27, Lns. 64-66). 

Regarding claim 3, as it depends from claim 2, Szeliski teaches the 
instructions from the user identify two or more substantially parallel 
features in the image panoramas (See Szeliski: Col. 20, Line 64 - Col. 2 1 , 
Line 6 and Lines 25-34). 

Regarding claim 5, as it depends from claim 2, Szeliski teaches the 
instructions from the user identify a horizon line of at least one image 
panorama (See Szeliski: Fig 4: Col. 9, Lns. 54-62). 

Regarding claim 6, as it depends from claim 2, Szeliski teaches the 
instructions comprise the identification of two or more areas of the image 
panoramas, each area containing one or more elements and further 
comprising automatically identifying the two elements contained in the 
two or more areas (See Szeliski: Fig. 6; Col. 20, Line 49 - Col. 21, Line 
24, wherein a feature-based point correspondence is established between 
a pair of images by dividing each image into patches and identifying 
prospective "feature" points within the patches.). 
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Regarding claim 8, as it depends from claim 1, the rationale of claim 1 is 
incorporated herein. The combination of Szeliski, Luken, and Seago 
substantially teach the invention as claimed. Specifically, Luken teaches 
the image panoramas are aligned relative to the reference coordinate 
system such that the directional vector of each panorama is at least 
substantially parallel to one axis of the reference coordinate system (See 
Luken: Fig. 7, Item 707; Col. 6, Line 40— Col. 7, Line 36). 

Regarding claim 9, as it depends from claim 1, the rationale of claim 1 is 
incorporated herein. The combination of Szeliski, Luken, and Seago 
substantially teach the invention as claimed. Specifically, Luken teaches 
the image panoramas are aligned relative to the reference coordinate 
system such that the directional vector of each panorama is at least 
substantially orthogonal to one axis of the reference coordinate system 
(See Luken: Fig. 6A and 7, Item 707; Col. 6, Line 40— Col. 7, Line 36, 
wherein since the six rectangular images are axis aligned, then a 
directional vector that is parallel to one axis must be perpendicular to 
the other two spatial axes. In other words, in order for a directional 
vector to intersect one of the rectangular images, that vector must be 
substantially parallel to one axis, which requires it to be substantially 
perpendicular to the others.). 
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Regarding claim 10, as it depends from claim 1, Szeliski teaches the 
image panoramas are aligned according to instructions received from a 
user (See Szeliski: Col. 27, Lns. 64-66, wherein the user aligns the 
image panoramas into texture space.). 

Regarding claim 39, as it depends from claim 1, Szeliski teaches the 
object is a room and the at least one boundary of the object is the 
intersection of a wall of the room with the floor (See Szeliski: Col. 27, 
Lns. 64-67, wherein the model can be a cube with six separate texture 
maps for each surface. Using an appropriate environment map would 
cause the top surface to be a ceiling, the bottom surface to be a floor, 
etc.) 

D. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Szeliski, Luken, and Seago, and in further view of Blank. 

Regarding claim 7, as it depends from claim 6, the combination of 

Szeliski, Luken and Seago does not expressly suggest using edge 
detection to automatically identify the two elements. Nevertheless, Blank 
teaches detecting the edges of an object and separates portions of the 
image that are outside the edge of the object (i.e., the background 
component) from portions of the image that are inside the edge (See 
Blank: Col. 4, Lns. 17-21). The two elements are therefore identified as 
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those elements within the edge, and those outside the edge. It would 
have been obvious to one of ordinary skill in the art, at the time the 
invention was made, to have used the edge detection methods, as taught 
by Blank, as an alternative to patch-based division, as taught by Szeliski, 
as modified by Luken, because it is an effective way to divide the image 
into smaller portions to conquer aligning all aspects of an image. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension 
of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire 
THREE MONTHS from the mailing date of this action. In the event a first 
reply is filed within TWO MONTHS of the mailing date of this final action 
and the advisory action is not mailed until after the end of the THREE- 
MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension 
fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date 
of the advisory action. In no event, however, will the statutory period for 
reply expire later than SIX MONTHS from the mailing date of this final 
action. 

Any inquiry concerning this communication or earlier communications 
from the examiner should be directed to Crystal Murdoch whose 
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telephone number is (571) 270-1043. The examiner can normally be 
reached on Mon. - Fri. 10:00 am to 6:30 pm. If attempts to reach the 
examiner by telephone are unsuccessful, the examiner's supervisor, Ulka 
Chauhan can be reached on (571) 272-7782. The fax phone number for 
the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from 

the Patent Application Information Retrieval (PAIR) system. Status 

information for published applications may be obtained from either 

Private PAIR or Public PAIR. Status information for unpublished 

applications is available through Private PAIR only. For more 

information about the PAIR system, see http://pair-direct.uspto.gov. 

Should you have questions on access to the Private PAIR system, contact 

the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you 

would like assistance from a USPTO Customer Service Representative or 

access to the automated information system, call 800-786-9199 (IN USA 

OR CANADA) or 571-272-1000. 

/Crystal Murdoch/ 
Examiner, Art Unit 2628 



/Ulka Chauhan/ 
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